Using Quasi-Experimental Data to Develop Empirical Generalizations for Persuasive Advertising

This paper argues that “quasi-experimental data” provide a valid and relatively low-cost approach toward developing empirical generalizations (EGs). These data are obtained from studies in which some key variables have been controlled in the design. These EGs are described as normative statements, i.e., “evidence-based principles.” Using data from 240 pairs of print advertisements from five editions of the Which Ad Pulled Best series, the authors analyzed 56 of the advertising principles (listed) from Persuasive Advertising by J. Scott Armstrong (New York: Palgrave Macmillan, forthcoming). These data controlled for target market, product, size of the advertisement, media, and in half the cases, for the brand. The advertisements differed, however, e.g. in illustrations, headlines, and text. The findings from the quasi-experimental analyses were consistent with field experiments for all seven principles where such comparisons were possible. Furthermore, for 26 principles they unanimously corroborated the available laboratory experiments as well as the meta-analyses for seven principles. In short, the quasi-experimental findings always agreed with experimental findings, even though the quasi-experimental analyses, and some of the experimental analyses, involved small samples, and often used different criteria. From an issue of JAR devoted to `empirical generalisations’: the papers were first presented at a conference at the Wharton School, University of Pennsylvania in December 2008. Comments Suggested Citation: Armstrong, J.S. and Patnaik, S. (2009). Using Quasi-Experimental Data to Develop Empirical Generalizations for Persuasive Advertising. Journal of Advertising Research. Vol. 49(2). p. 170-175. The definitive version of this article is available at: http://www.journalofadvertisingresearch.com/ ArticleCenter/default.asp?ID=89489&Type=Article This journal article is available at ScholarlyCommons: http://repository.upenn.edu/marketing_papers/167 JAR49(2) 09-023 1/6 05/19/09 8:08 am REVISED PROOF Page:170 Using Quasi-Experimental Data To Develop Empirical Generalizations For Persuasive Advertising "Quasi-experimental data" provide a valid and relatively low-cost approach toward developing empirical generalizations (EGs). These data are obtained from studies in which some key variables have been controlled in the design. We describe our EGs as "evidence-based principles." Using data from 240 pairs of print advertisements from five editions of the Which Ad Pulled Best series, we analyzed 56 of the advertising principles from Persuasive Advertising by J. Scott Armstrong (New York: Palgrave Macmillan, forthcoming). These data controlled for target market, product, size of the advertisement, media, and in half the cases, for brand. Aspects of the advertisements differed, however, as in illustrations, headlines, color, and text. The findings from quasi-experimental analyses were consistent with field experiments for all seven principles where such comparisons were possible. Furthermore, for 26 principles they unanimously corroborated the available laboratory experiments as well as the meta-analyses for seven principles. In short, quasi-experimental findings always agreed with experimental findings. This is impressive given that the quasi-experimental analyses—and some of the experimental analyses—involved small samples, and often used different criteria. J. SCOTT ARMSTRONG The Wharton School armstrong@wharton. upenn.edu SANDEEP PATNAIK Gallup and Robinson sandeep.patnaik@ gallup-robinson.com INTRODUCTION Progress in advertising depends upon the use of cumulative knowledge. This knowledge can be transformed into empirical generalizations (EGs). In this article, we discuss a type of EG that we refer to as "evidence-based principles," or, more simply, as principles. We describe the types of evidence that can be used to develop principles. We then describe the quasi-experimental data on full-page print advertisements that we used to assess advertising principles. Finally, we examine the validity of the analyses of these data by comparing the findings with those from other types of studies including laboratory experiments, field experiments, and meta-analyses based on experimental evidence. SOURCES OF EVIDENCE Nonexperimental studies involve analyses of data on a set of advertisements to determine which variables are most closely related to success. This is probably the dominant way by which people draw inferences about the value of persuasive strategies in advertising. In quasi-experimental studies, the design controls for some (but not all) variables when comparing different advertisements. For example, these studies might compare advertisements for a given brand and media. Other variables, however, may not be controlled. This approach is seldom used to analyze persuasive advertising. In experimental studies, the values of all key causal variables are controlled. For example, a researcher who wants to know whether humor is DOI: 10.2501/S0021849909090230 170 JOURNAL OF ADVERTISING RESEARCH June 2009 JAR49(2) 09-023 2/6 05/19/09 8:08 am REVISED PROOF Page:171 DEVELOPING EGs FOR PERSUASIVE PRINT ADVERTISING In some cases, the advertisements were coded by more than one person. For example, advertisements for one principle, "Communicate a Unique Selling Proposition (USP)," were coded by three coders to establish that the coding related to a USP and not merely to a benefit. The coders were asked to keep notes regarding uncertainty or other observations relevant to their coding decision. Coding sheets were prepared (sample sheets are provided at advertisingprinciples.com). All coding was done prior to viewing the criteria, except for two principles in the early stages, when the procedure was being developed. The process meant that no coders had prior knowledge regarding an advertisement's degree of effectiveness. In a number of instances, the advertisements were coded several times to examine the main principle and the conditions associated with the principle. For example, while coding for the principle "Communicate a USP," our initial analysis yielded 18 pairs in which one advertisement in the pair claimed a USP while the other did not make such a claim. Next, all advertisements with USP claims were carefully coded to check whether their USPs could have been claimed by other products or services. We found 27 pairs of advertisements where one of the advertisements had a USP that could not have been claimed by any other product while the other advertisement stated a common USP. In general, advertisements with USPs that were not common were better recalled than advertisements using a "common" USP. For example, an advertisement for Norland Bone Desensitometer that said "Unique design of the scanner allows scans anywhere on the surface of the table, and at any angle. Your patient does not move, the scanner arm does. . . . No other system offers this feature" had a recall thrice that for another EMPIRICAL GENERALIZATION Communicate a unique selling proposition. Ideally, it should be based on an important benefit; alternatively, and riskier, it could be based on a feature that clearly implies a benefit. It is effective if it is unique in the minds of consumers even though other brands could make the same claim. However, it is especially effective if it cannot be easily matched by competitors. This generalization, previously regarded as problematic, is supported by recent experimental evidence. persuasive creates two identical advertisements, except that one advertisement is humorous and the other is not. The researcher shows the advertisements to randomly assigned subjects in identical situations and records their reactions. Experimental evidence comes from two types of studies: laboratory experiments and field experiments. The former allows for tighter control, while the latter adds realism. The WAPB database allows testing of the direction and effect size for the advertising principles because most of the conditions were identical for each advertisement in the pairs. The target market, product, and media were the same. Of the 240 pairs of advertisements, 123 were paired against the same brand. Aspects of the advertisements differed, however, such as in illustrations, headlines, color and text. Instructors' guides for each WAPB edition provided recall scores for advertisements in 40 of the pairs and reader-interest scores for 10 pairs. (Reader-interest scores were not included in the 9th edition.) We consider both of these scores under the term "recall." Recall scores were based on a subject's ability to identify both the product and advertiser the day following exposure. ANALYSIS OF THE WAPB QUASI-EXPERIMENTAL DATABASE We analyzed full-page print advertisements from Which Ad Pulled Best (WAPB) 5th through 9th editions (Burton and Purvis, 1987-2002). Each edition contains 50 pairs of advertisements (except for the 9th edition, which has 40 pairs). These advertisements had been prepared by leading U.S. advertisers and were then sent for testing by Gallup & Robinson. In effect, they represent quasi-field experiments. The advertisements featured in WAPB 5th through 8th editions are archived at www.advertisingprinciples. com. WAPB 9th edition is archived at www.gallup-robinson.com Coding the advertisements A team of research assistants coded the conditions and actions for each advertisement. In some cases (i.e., "How many words are in a headline"), the coding was straightforward. In many cases, however, the coding was subjective (i.e., determining whether an advertisement related to a high-involvement or low-involvement product). Someone familiar with the principle, therefore, had to be charged with thoughtfully doing the coding. The coders, however, had no knowledge of the purpose of the coding. Criteria The WAPB criteria for these quasiexperimental data relate to recall. Recall is an intermediate step; if you are not aware of an advertisement, you are unlikely to act on it. Zinkhan and Gelb (1986) found that high "noted" scores were positively related to purchase intentions (r = 0.52). JOURNAL OF ADVERTISING RESEARCH 171 June 2009 JAR49(2) 09-023 3/6 05/19/09 8:08 am DEVELOPING EGs FOR PERSUASIVE PRINT ADVERTISING REVISED PROOF Page:172 TABLE 1 Quasi-Experimental Results on Recall Gain for Persuasion Principles Number of Recall Advertising ................................................................................................................................................................................................................................................ ................................................................................................................................................................................................................................................ ................................................................................................................................................................................................................................................ ................................................................................................................................................................................................................................................ ................................................................................................................................................................................................................................................ ................................................................................................................................................................................................................................................ ............................................................. ................................................................. 1.65 10 Include brand and company names (double-branding). 1.71 21 Describe a problem and show how the product solves it. 1.73 10 Make the first paragraph relevant 1.74 46 Communicate a Unique Selling Proposition (USP) (one that cannot be claimed by other brands). 2.04 45 Offer verifiable evidence. 2.12 6 Description of Principles: Includes Necessary Conditions Pairs Gain .................................................................................................................. Consider celebrity endorsements for gaining attention. ................................................................................................................................................................................................................................................ ................................................................................................................................................................................................................................................ ................................................................................................................................................................................................................................................ ................................................................................................................................................................................................................................................ ................................................................................................................................................................................................................................................ ................................................................................................................................................................................................................................................ ................................................................................................................................................................................................................................................ ................................................................................................................................................................................................................................................ ................................................................................................................................................................................................................................................ ................................................................................................................................................................................................................................................ ................................................................................................................................................................................................................................................ ................................................................................................................................................................................................................................................ ................................................................................................................................................................................................................................................ ................................................................................................................................................................................................................................................ ................................................................................................................................................................................................................................................ ................................................................................................................................................................................................................................................ ................................................................................................................................................................................................................................................ 37 Provide a reason. ................................................................................................................................................................................................................................................ 16 Use a typeface to enhance meaning. ................................................................................................................................................................................................................................................ 58 Use informative subheadings for long copy. ................................................................................................................................................................................................................................................ 24 Use wordplay if it is clearly related to the product. ................................................................................................................................................................................................................................................ 26 Avoid negative words unless the target market believes the opposite. ................................................................................................................................................................................................................................................ (continued) 1.23 1.25 1.25 1.28 1.29 Provide evidence that the product is widely used. 1.31 7 Use familiar words. 1.32 14 Use concrete words. 1.32 38 When using metaphors, make them concrete. 1.34 37 Provide product information that customers need. 1.34 37 Use product-related questions only if you have good answers. 1.39 12 Make elements of an advertisement reinforce one another. 1.42 12 Show the product. 1.43 14 Make the brand or company name prominent if it conveys a good image. 1.44 18 Consider mystery advertisements only when the brand or organization does not enhance the claim. 1.44 22 For high-involvement products, the reasons should be strong. 1.48 25 Include the brand name in the headline. 1.49 24 Balance the layout. 1.50 36 Use descriptive headlines for high-involvement products. 1.52 24 Illustrations should support the basic message. 1.54 43 Use positive arguments. 1.60 24 Provide news, but only if it is real. 1.64 20 172 JOURNAL OF ADVERTISING RESEARCH June 2009 JAR49(2) 09-023 4/6 05/19/09 8:08 am REVISED PROOF Page:173 DEVELOPING EGs FOR PERSUASIVE PRINT ADVERTISING